Local Geometric Structure Of Data Research Articles

One-step multi-view clustering via deep-level semantics exploiting

Multi-view clustering (MVC) has gained promising performance improvement compared with traditional signal-view clustering due to the complementary information of multiple views. However, existing MVC methods exploit clustering structure by utilising signal-layer mapping, such that they cannot exploit the underlying deep-level semantic information in complex and interleaved multi-view data. Moreover, existing methods usually conduct multi-view fusion and clustering separately, which results in unpromising performance. To address the above problems, one-step MVC via deep-level semantics exploiting (DLSE) is proposed to exploit deep-level semantic information and learn the indicator matrix using a one-step manner. To be specific, a novel deep matrix factorisation (DMF) paradigm is designed to exploit the hierarchical semantics via a layer-wise scheme, so that samples from the same clusters are forced to be closer in the low-dimensional space layer by layer. Furthermore, to make the learned representation preserve the local geometric structure of data, DLSE introduces a local preservation regularisation to guide DMF. Meanwhile, by employing spectral rotating fusion, the cluster indicator can be obtained directly. Extensive experiments demonstrate the superiority of DLSE in contrast with some state-of-the-art methods.

Joint Sparse Locality Preserving Regression for Discriminative Learning

Ridge Regression (RR) is a classical method that is widely used in multiple regression analysis. However, traditional RR does not take the local geometric structure of data into consideration for discriminative learning and it is sensitive to outliers as it is based on <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$L_{2}$</tex-math></inline-formula> -norm. To address this problem, this article proposes a novel method called Joint Sparse Locality Preserving Regression (JSLPR) for discriminative learning. JSLPR not only applies <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$L_{2,1}$</tex-math></inline-formula> -norm on both loss function and regularization term but also takes the local geometric structure of the data into consideration. The use of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$L_{2,1}$</tex-math></inline-formula> -norm can guarantee the robustness to outliers or noises and the joint sparsity for effective feature selection. Taking the local geometric structure into consideration can improve the performance of the feature extraction and selection method when the data lie on a manifold. To solve the optimization problem of JSLPR, an iterative algorithm is proposed and the convergence of the algorithm is also proven. Experiments on four famous face databases are conducted and the results show the merit of the proposed JSLPR on feature extraction and selection.

Deep low-rank tensor embedding for multi-view subspace clustering

Despite the good clustering performance, most of existing multi-view subspace clustering methods fail to consider the non-linear relationships of high-dimensional data, higher-order correlations of different views, and optimal local geometric structure of the latent embedding space simultaneously. To this end, we put forward a novel multi-view subspace clustering model, named deep low-rank tensor embedding (DLTE). In DLTE, we project the high-dimensional data into the low-dimensional embedding space by utilizing deep non-negative matrix factorization (NMF), which can efficiently capture the complex non-linear relationship of data. Moreover, in the deep embedding space, we utilize the weighted low-rank tensor constraint to capture the global structure and higher-order correlations of different views while considering the contributions of different views. Additionally, we introduce an optimal graph Laplacian to learn a more reasonable local geometric structure of data in the deep embedding space. At last, comprehensive experimental results on eleven datasets indicate the superiority and effectiveness of the proposed DLTE method.

Unsupervised feature selection via discrete spectral clustering and feature weights

Most of the existing unsupervised feature selection methods learn the cluster structure through spectral clustering, and then use various regression models to introduce the data matrix into the indicator matrix to obtain feature selection matrix. In these methods, the clustering indicator matrix is usually continuous value, which is not the best choice for the matrix in terms of its supervising role in feature selection. Based on this, unsupervised feature selection via discrete spectral clustering and feature weights (FSDSC) is proposed in this paper. First, FSDSC integrates regression model and spectral clustering in a unified framework for feature selection, and introduces a feature weight matrix, which intuitively expresses the importance of each feature with its diagonal elements. Compared with the common feature selection matrix that requires constraints such as sparse regular items, the appearance of the feature weight matrix reduces the complexity of the model and simplifies the calculation process of feature evaluation. Secondly, for the value of the indicators matrix, the spectral clustering is improved to obtain a discrete clustering indicator matrix, which provides clearer guidance information for feature selection. Finally, in order to avoid trivial solutions, the transformation matrix is constrained by orthogonal constraint. The combination of the orthogonal regression model and spectral clustering enables the algorithm to perform feature selection and manifold information learning at the same time, thereby preserving the local geometric structure of data. Compared with other excellent unsupervised feature selection algorithms, the experimental results prove the effectiveness of the proposed algorithm.

Fault monitoring of batch process based on multi-stage optimization regularized neighborhood preserving embedding algorithm

Batch process is an important type of industrial production process, and the process mechanism is complex. It is difficult to accurately describe the dynamic changes of the production process of multi-stage time-varying batch process. In addition, the data of batch process contain not only global information but also local information. The traditional neighborhood preserving embedded algorithm is used to maintain the local geometric structure of data while ignoring the global information, and the extracted latent variables cannot fully characterize batch process. Therefore, we propose a multi-stage optimization regularized neighborhood preserving embedding (ORNPE) algorithm. First, the multiple process stages are separated by affinity propagation (AP) algorithm. Second, based on maintaining local information of neighborhood preserving embedding algorithm, slow feature analysis algorithm is used to extract dynamic time-varying global information. Then, cross-entropy is used to optimize the global information, and the extraction ability of the global information is improved. Finally, a monitoring index based on support vector data description is constructed to eliminate adverse effects of non-Gaussian data for monitoring performance. The effectiveness and advantages of the proposed algorithm based on monitoring strategy are illustrated by the penicillin fermentation process and a semiconductor industry process.

Feature selection based on non-negative spectral feature learning and adaptive rank constraint

Unsupervised feature selection plays a significant role in data classification and clustering. General regression models cannot directly exploit the information on the feature space and fail to accurately describe the local geometric structure of data during feature selection. To address these problems, this paper proposes the feature selection algorithm, which is based on non-negative spectral feature learning and adaptive rank constraint (NNSAFS). First, the algorithm utilizes the residual term in sparse regression to ensure that the learned low-dimensional subspaces have greater fault tolerance and introduces a feature graph on the sparse transformation matrix to reveal the manifold information on the feature space. This sparse transformation matrix is the projection matrix because the introduction of feature graphs on this matrix can connect manifold learning to feature selection. In addition, traditional spectral clustering algorithms usually construct fixed similarity graphs for clustering analysis. The NNSAFS algorithm imposes a rank constraint on the clustering indicator matrix, which is equivalent to the graph regularization that can recover accurate local structure information. Moreover, the similarity matrix in the regularization term is constructed by using the maximum entropy theory, which can increase the adaptability of manifold learning. Finally, the algorithm imposes the l1-norm constraint on the projection matrix, which makes the selected features more conducive to clustering performance. The clustering performance of the NNSAFS algorithm is evaluated against seven other unsupervised feature selection algorithms on nine benchmark datasets. The experimental results show that the features selected by the proposed algorithm are more discriminative and outperform other algorithms in the clustering task.11The code of the proposed algorithm has been uploaded and available at the following link: https://github.com/Vita427/NNSAFS.git.

Low-Rank Joint Embedding and Its Application for Robust Process Monitoring

Industrial data are in general corrupted by noises and outliers. In this context, robustness to the contaminated data is a challenging issue in process monitoring. In this article, a novel method named low-rank joint embedding is proposed for robust process monitoring. By learning a low-rank coefficient matrix, low-rank joint embedding can capture the global structure of the original data and alleviate the negative effect of outliers, making the monitoring results more reliable. Moreover, a manifold regularization is introduced to preserve the local geometric structure of data, which enables the extracted low-dimensional representation of data to be more faithful and informative to enhance the monitoring capability. Based on projection learning, the low-rank joint embedding can learn an explicit projection that transforms the data not involved in the training data into the low-dimensional space, avoiding the out-of-sample problem. Furthermore, a reconstruction-based contribution plots based on the low-rank joint embedding is developed to identify the potential faulty variables. Case studies on the Tennessee Eastman process and a real industrial application demonstrate the effectiveness of the proposed monitoring approach.

Multi-label feature selection with constraint regression and adaptive spectral graph

Like single-label learning, multi-label learning also suffers from the curse of dimensionality. Due to the existence of high-dimensional data, feature selection as a preprocessing tool always plays a key position in the multi-label learning. Although a variety of multi-label feature selection methods have been developed, they neglect to consider the redundancy of features, thereby degrading learning performance. To address this problem, we present a novel multi-label feature selection approach with uncorrelated regression and adaptive spectral graph. Specifically, we first construct a manifold framework with uncorrelated regression model to hunt for uncorrelated yet discriminative features, which also utilize the low-dimensional representation based on feature space to fit the label distribution. Then, a spectral graph term based on information entropy is incorporated into the manifold framework, so as to ensure the local geometric structure of data in subsequent learning process. Following the above principles, we design an objective function to achieve multi-label feature selection, and propose a detailed optimization method. Comprehensive experiments are conducted on multiple public multi-label data sets, the results show that our method is superior to other compared methods.

Joint Nonnegative Matrix Factorization Based on Sparse and Graph Laplacian Regularization for Clustering and Co-Differential Expression Genes Analysis

The explosion of multiomics data poses new challenges to existing data mining methods. Joint analysis of multiomics data can make the best of the complementary information that is provided by different types of data. Therefore, they can more accurately explore the biological mechanism of diseases. In this article, two forms of joint nonnegative matrix factorization based on the sparse and graph Laplacian regularization (SG-jNMF) method are proposed. In the method, the graph regularization constraint can preserve the local geometric structure of data.L2,1-norm regularization can enhance the sparsity among the rows and remove redundant features in the data. First, SG-jNMF1 projects multiomics data into a common subspace and applies the multiomics fusion characteristic matrix to mine the important information closely related to diseases. Second, multiomics data of the same disease are mapped into the common sample space by SG-jNMF2, and the cluster structures are detected clearly. Experimental results show that SG-jNMF can achieve significant improvement in sample clustering compared with existing joint analysis frameworks. SG-jNMF also effectively integrates multiomics data to identify co-differentially expressed genes (Co-DEGs). SG-jNMF provides an efficient integrative analysis method for mining the biological information hidden in heterogeneous multiomics data.

Low-Rank Discriminative Adaptive Graph Preserving Subspace Learning

The global and local geometric structures of data play a key role in subspace learning. Although many manifold-based subspace learning methods have been proposed for preserving the local geometric structure of data, they usually use a predefined neighbor graph to characterize it. However, the predefined neighbor graph might be not optimal since it keeps fixed during the subsequent subspace learning process. Moreover, most manifold-based subspace learning methods ignore the global structure of data. To address these issues, we propose a low-rank discriminative adaptive graph preserving (LRDAGP) subspace learning method for image feature extraction and recognition by integrating the low-rank representation , adaptive manifold learning, and supervised regularizer into a unified framework. To capture the optimal local geometric structure of data for subspace learning, LRDAGP adopts an adaptive manifold learning strategy that the neighbor graph is adaptively updated during the subspace learning process. To capture the optimal global structure of data for subspace learning, LRDAGP also seeks the low-rank representations of data in a low-dimensional subspace during the subspace learning process. Moreover, for improving the discrimination ability of the learned subspace, a supervised regularizer is designed and incorporated into the LRDAGP model. Experimental results on several image datasets show that LRDAGP is effective for image feature extraction and recognition.

Cross-Dataset Transfer Driver Expression Recognition via Global Discriminative and Local Structure Knowledge Exploitation in Shared Projection Subspace

Facial expression is one of the important characteristics of drivers during driving. It is very useful in safe driving detection. Recognizing drivers’ expressions by the facial images can be solved with machine learning classification strategies. To obtain a reliable reorganization performance, most of approaches assume that the facial images in the training and testing datasets are independently and identically distributed. However, for real time drivers’ facial expression recognition, due to vehicle motion, changes in illumination, noise and head movement, the features displayed for the training dataset may be not valid for the testing dataset. To solve this problem, a novel approach is proposed for cross-dataset transfer driver expression recognition via global discriminative and local structure knowledge exploitation in shared projection subspace (GD-LS-SS). By leaning a shared common subspace, GD-LS-SS utilizes the local geometrical structure of data by exploiting the knowledge of graph topology, meanwhile exploiting the global discriminative information by using the pairwise constrained knowledge between the source and labeled target data. Taking advantage of kernel trick, the kernel version of GD-LS-SS is proposed to learn the kernel projection for handling nonlinear cross-dataset transfer and to further promote the recognition accuracy. Experiments on the KMU-FED dataset show that the satisfactory recognition performance of GD-LS-SS outperforms several traditional non-transfer and related transfer approaches.

A generalized least-squares approach regularized with graph embedding for dimensionality reduction

In current graph embedding methods, low dimensional projections are obtained by preserving either global geometrical structure of data or local geometrical structure of data. In this paper, the PCA (Principal Component Analysis) idea of minimizing least-squares reconstruction errors is regularized with graph embedding, to unify various local manifold embedding methods within a generalized framework to keep global and local low dimensional subspace. Different from the well-known PCA method, our proposed generalized least-squares approach considers data distributions together with an instance penalty in each data point. In this way, PCA is viewed as a special instance of our proposed generalized least squares framework for preserving global projections. Applying a regulation of graph embedding, we can obtain projection that preserves both intrinsic geometrical structure and global structure of data. From the experimental results on a variety of face and handwritten digit recognition, our proposed method has advantage of superior performances in keeping lower dimensional subspaces and higher classification results than state-of-the-art graph embedding methods.

Dual graph regularized compact feature representation for unsupervised feature selection

As one of the dimensionality reduction methods, unsupervised feature selection aims to select a subset of original features without the labels of data instances. It becomes more and more important since large amounts of unlabelled high dimensional data need to be processed in many machine learning and data mining tasks. In this paper, we propose a novel unsupervised feature selection method via compact feature representation, with the local geometrical structure of data being preserved by using a dual Laplacian graph regularization term. In detail, different to many previous representation based methods which use the original data as representation dictionary, we propose to learn a feature dictionary subspace for compact and robust feature representation. During the dictionary learning process, the l2,1-norm is imposed on the combination coefficient matrix to select discriminate features for constructing the latent feature dictionary. Meanwhile, the local geometrical structure of original data is preserved from the perspectives of subspace learning and feature representation. In general, our method conducts dictionary learning and unsupervised feature selection simultaneously. We develop an efficient optimization algorithm based on Alternating Direction Method of Multipliers to solve the proposed optimization problem and experiments on various types of real world datasets are conducted to demonstrate the effectiveness of the proposed method.

Neighborhood preserving neural network for fault detection

A novel statistical feature extraction method, called the neighborhood preserving neural network (NPNN), is proposed in this paper. NPNN can be viewed as a nonlinear data-driven fault detection technique through preserving the local geometrical structure of normal process data. The “local geometrical structure ” means that each sample can be constructed as a linear combination of its neighbors. NPNN is characterized by adaptively training a nonlinear neural network which takes the local geometrical structure of the data into consideration. Moreover, in order to extract uncorrelated and faithful features, NPNN adopts orthogonal constraints in the objective function. Through backpropagation and eigen decomposition (ED) technique, NPNN is optimized to extract low-dimensional features from original high-dimensional process data. After nonlinear feature extraction, Hotelling T2 statistic and the squared prediction error (SPE) statistic are utilized for the fault detection tasks. The advantages of the proposed NPNN method are demonstrated by both theoretical analysis and case studies on the Tennessee Eastman (TE) benchmark process. Extensive experimental results show the superiority of NPNN in terms of missed detection rate (MDR) and false alarm rate (FAR). The source code of NPNN can be found in https://github.com/htzhaoecust/npnn.

Graph regularized low-rank tensor representation for feature selection

Recently, considerable efforts have been made in feature selection to improve the original feature subspace. In this paper, we proposed a graph regularized low-rank tensor representation (GRLTR) for feature selection. We jointly incorporated the low-rank representation and the graph embedding into a unified learning framework to preserve the intrinsic global low-dimension structure and local geometrical structure of data together. According to the wide presence of multidimensional data, our proposed framework is based on tensor, which can faithfully maintain the information. To improve the performance of specific clustering task, we employed the idea of embedded-based feature selection into our model for optimizing the feature representation and clustering result simultaneously. Experimental results on six available datasets suggest our proposed approach produces superior performances compared with several state-of-the-art methods.

Generalized Uncorrelated Regression with Adaptive Graph for Unsupervised Feature Selection.

Unsupervised feature selection always occupies a key position as a preprocessing in the tasks of classification or clustering due to the existence of extra essential features within high-dimensional data. Although lots of efforts have been made, the existing methods neglect to consider the redundancy of features, and thus select redundant features. In this brief, by virtue of a generalized uncorrelated constraint, we present an improved sparse regression model [generalized uncorrelated regression model (GURM)] for seeking the uncorrelated yet discriminative features. Benefited from this, the structure of data is kept in the Stiefel manifold, which avoids the potential trivial solution triggered by a conventional ridge regression model. Besides that, the uncorrelated constraint equips the model with the closed-form solution. In addition, we also incorporate a graph regularization term based on the principle of maximum entropy into the GURM model (URAFS), so as to embed the local geometric structure of data into the manifold learning. An efficient algorithm is designed to perform URAFS by virtue of the existing generalized powered iteration method. Extensive experiments on eight benchmark data sets among seven state-of-the-art methods on the task of clustering are conducted to verify the effectiveness and superiority of the proposed method.

Robust unsupervised feature selection via dual self-representation and manifold regularization

Unsupervised feature selection has become an important and challenging pre-processing step in machine learning and data mining since large amount of unlabelled high dimensional data are often required to be processed. In this paper, we propose an efficient method for robust unsupervised feature selection via dual self-representation and manifold regularization, referred to as DSRMR briefly. On the one hand, a feature self-representation term is used to learn the feature representation coefficient matrix to measure the importance of different feature dimensions. On the other hand, a sample self-representation term is used to automatically learn the sample similarity graph to preserve the local geometrical structure of data which has been verified critical in unsupervised feature selection. By using l2,1-norm to regularize the feature representation residual matrix and representation coefficient matrix, our method is robustness to outliers, and the row sparsity of the feature coefficient matrix induced by l2,1-norm can effectively select representative features. During the optimization process, the feature coefficient matrix and sample similarity graph constrain each other to obtain optimal solution. Experimental results on ten real-world data sets demonstrate that the proposed method can effectively identify important features, outperforming many state-of-the-art unsupervised feature selection methods in terms of clustering accuracy (ACC) and normalized mutual information (NMI).

Robust graph regularized unsupervised feature selection

Recent research indicates the critical importance of preserving local geometric structure of data in unsupervised feature selection (UFS), and the well studied graph Laplacian is usually deployed to capture this property. By using a squared l2-norm, we observe that conventional graph Laplacian is sensitive to noisy data, leading to unsatisfying data processing performance. To address this issue, we propose a unified UFS framework via feature self-representation and robust graph regularization, with the aim at reducing the sensitivity to outliers from the following two aspects: i) an l2, 1-norm is used to characterize the feature representation residual matrix; and ii) an l1-norm based graph Laplacian regularization term is adopted to preserve the local geometric structure of data. By this way, the proposed framework is able to reduce the effect of noisy data on feature selection. Furthermore, the proposed l1-norm based graph Laplacian is readily extendible, which can be easily integrated into other UFS methods and machine learning tasks with local geometrical structure of data being preserved. As demonstrated on ten challenging benchmark data sets, our algorithm significantly and consistently outperforms state-of-the-art UFS methods in the literature, suggesting the effectiveness of the proposed UFS framework.

Kernelized sparse hashing for scalable image retrieval

Recently, hashing has been widely applied to large scale image retrieval applications due to its appealing query speed and low storage cost. The key idea of hashing is to learn a hash function that maps high dimensional data into compact binary codes while preserving the similarity structure in the original feature space. In this paper, we propose a new method called the Kernelized Sparse Hashing, which generates sparse hash codes with ℓ1 and non-negative regularizations. Compared to traditional hashing methods, our method only activates a small number of relevant bits on the hash code and hence provides a more compact and interpretable representation of data. Moreover, the kernel trick is introduced to capture the nonlinear similarity of features, and the local geometrical structure of data is explicitly considered in our method to improve the retrieval accuracy. Extensive experiments on three large-scale image datasets demonstrate the superior performance of our proposed method over the examined state-of-the-art techniques.

Global and Local Structure Preservation for Feature Selection

The recent literature indicates that preserving global pairwise sample similarity is of great importance for feature selection and that many existing selection criteria essentially work in this way. In this paper, we argue that besides global pairwise sample similarity, the local geometric structure of data is also critical and that these two factors play different roles in different learning scenarios. In order to show this, we propose a global and local structure preservation framework for feature selection (GLSPFS) which integrates both global pairwise sample similarity and local geometric data structure to conduct feature selection. To demonstrate the generality of our framework, we employ methods that are well known in the literature to model the local geometric data structure and develop three specific GLSPFS-based feature selection algorithms. Also, we develop an efficient optimization algorithm with proven global convergence to solve the resulting feature selection problem. A comprehensive experimental study is then conducted in order to compare our feature selection algorithms with many state-of-the-art ones in supervised, unsupervised, and semisupervised learning scenarios. The result indicates that: 1) our framework consistently achieves statistically significant improvement in selection performance when compared with the currently used algorithms; 2) in supervised and semisupervised learning scenarios, preserving global pairwise similarity is more important than preserving local geometric data structure; 3) in the unsupervised scenario, preserving local geometric data structure becomes clearly more important; and 4) the best feature selection performance is always obtained when the two factors are appropriately integrated. In summary, this paper not only validates the advantages of the proposed GLSPFS framework but also gains more insight into the information to be preserved in different feature selection tasks.